When will Feature Feedback help? Quantifying the Complexity of Classification Problems

Supervised learning typically requires human effort to label a large number of training instances. Active learning strives to decrease the number of labeled training examples needed by actively engaging the learner and the human in an interactive process. Active learning has proven to be effective in many domains. With few training examples, past work has found that user prior knowledge on the importance of features, or interactive feature feedback, can guide the learner to converge faster, that is, with lower labeling costs. In this paper we aim to understand the kinds of problems for which such extra feedback are significantly beneficial. In other words, we ask what kind of problems can significantly benefit from interactive learning and whether for some problems the user has no choice but to engage in the tedious process of labeling many examples. Towards this goal, we define a set of four difficulty measures, 2 each of instance and feature complexity, for linear classification problems. These measures can efficiently be computed for real world problems for which linear classifiers are effective, such as text classification. We quantify the difficulty of 358 text classification problems and 9 corpora using our measures, illustrating the spectrum of problems that exist in text classification in addition to quantifying results that have only been qualitatively discussed in the text classification literature. We verify the intimate relationship (a high positive correlation) between feature complexity and instance complexity using our measures. We then use these measures to understand when feature feedback is likely to be very useful. We observe that many problems in the commonly used data sets are of low to medium complexity, that is, only roughly 10s of well selected features are required to gain most of the maximum attained performance on such concepts. We find that learning these kinds of problems especially stands to benefit from feature feedback. We note that our empirical difficulty measures and the rankings of problems and domains are of independent interest, beyond the active learning setting.